In neural networks, memory matrixes are defined as being full, which are generally considered to be between 20-30% up to 100% full of non-zero data, and sparse matrixes, which are defined as being less than 20-30% full. A computer memory matrix may be a 1,000.times.1,000 array, which is capable of holding 1,000,000 words of data. In the case of a sparse matrix, the memory may only be 5% full, containing only 50K words of actual data.
In the case of a neural network type of processor node, the data stored in memory is cycled on a clock, or cycle, basis, with every memory location being acted upon by processor units which manipulate data. Particularly in the case of a matrix that is only 5 to 10% full, a great deal memory space is "filled" with zero information. From an efficiency standpoint, the occupation of hundreds of thousands of memory addresses with zero information is not acceptable. Memory is probably the most expensive component of computer technology.
There are several important practical problems that cannot be solved using existing, conventional algorithms executed by traditional, conventional computers. These problems are often incompletely specified and are characterized by many weak constraints requiring large search spaces.
The processing of primary cognitive information by computers, such as computer speech recognition, computer vision, and robotic control, fall into this category. Traditional computational models bog down to the point of failure under the computational load if they are tasked to solve these types of problems. Yet animals perform these tasks using neurons that are millions of times slower than transistors. Feldman's 100-step rule states that a "human" cognitive process having a time of 500 msec. can be accomplished in 5 msec neuron switching time. This implies that there are two vastly different computational models at work. It also suggests that in order to build computers that will do what nervous systems do, the computers should be structured more like nervous systems.
A nervous system, and a neurocomputational computer, is characterized by continuous, non-symbolic, and massively parallel structure that is fault-tolerant of input noise and hardware failure. Representations, ie, the input, is distributed among groups of computing elements, which independently reach a result or conclusion, and which then generalize and interpolate information to reach a final output conclusion.
Put another way, connectionist/neural networks search for "good" solutions using massively parallel computations of many small computing elements. The model is one of parallel hypothesis generation and relaxation to the dominant, or "most-likely" hypothesis. The search speed is more or less independent of the size of the search space. Learning is a process of incrementally changing the connection (synaptic) strengths, as opposed to allocating data structures. "Programming" in such a neural network is by example.